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ABSTRACT 

Since both cost /quality goals and production en- 
vironments differ, this study presents an approach for 
customizing a characteristic set of software metrics to an 
environment. The approach Is applied In the Software 
Engineering Laboratory (SEL), a NASA Goddard produc- 
tion environment, to 49 candidate process and product 
metrics of 652 modules from six (51,000 - 112,000 line) 
projects. For this particular environment, the method 
yielded the characteristic metric set {source lines, fault 
correction effort per executable statement, design effort, 
code effort, number of I/O parameters, number of ver- 
sions}. The uses examined for a characteristic metric set 
Include forecasting the effort for development, 
modification, and fault correction of modules based on 
historical data. 


1. Introduction 

Several metrics have been proposed to predict pro- 
duct cost/quallty and to capture distinct project aspects 
(8, 12, 18, 19, 21]. The effectiveness of the metrics In 
capturing what Is Intended, however, has depended on 
the particular environment examined [1, 4, 9, 10, 13, 17, 
27, 28, 20], A particular software metric that has been 
useful to characterize, evaluate, or predict aspects of soft- 
ware development In one environment may have limited 
usefulness elsewhere. The differing cost/quallty goals 
among environments and the diversity In methodology, 
software type, etc. contribute to the Inconsistent perfor- 
mance of metrics. Thus, It seems Inappropriate to at- 
tempt to select a set of software metrics that have 
universal effectiveness across all software environments. 
The selection of a set of metrics appropriate for a partic- 
ular environment must consider Its Individual features; 
that Is, a metric set must be customized to a specific en- 
vironment. 

Section 2 describes the Idea of characteristic soft- 
ware metric sets. Section 3 presents an approach for cus- 
tomizing a characteristic set of cost and quality metrics 
to an environment. The application of the approach In a 
software production environment Is discussed In Section 
4. Section 5 Investigates the use of a characteristic 
metric set as a management tool. Section 6 presents the 
conclusions from this work. 


2. Characteristic Software Metric Sets 

The successful management of software projects 
requires a diverse range of capabilities. Including monitor- 
ing and controlling the evolving software system and 
forecasting the outcome of the development. Techniques 
that assist In these management functions may lead to 
more successful projects, and possibly higher product re- 
quirement conformance and operational reliability. • The 
Idea of a characteristic software metric set supports 
several aspects of software management. 

A characteristic software metric set Is a concise 
collection of metrics that capture distinct factors In a 
software' development/maintenance environment. A 
characteristic metric set can be thought of as a vector of 
metrics that represents different areas of Importance In an 
environment. Since both cost/quallty goals and produc- 
tion environments differ, the particular factors that are 
captured by the metrics in the set will differ across en- 
vironments. The calculation of a characteristic metric set 
should be based on the particular cost and quality goals 
In an environment, and reflect the Inherent differences of 
an environment from others. 

A characteristic metric set may be used to 1) 
characterize an environment, 2) compare aD environment 
with others, 3) monitor current project status, or 4) fore- 
cast project outcome relative to past projects, when 
metrics in the set are available early In development. 
Once the distinct factors In an environment’s set are 
determined, the set then characterizes what aspects are 
Important In the environment. Comparing the charac- 
teristic set of factors In one environment with the sets of 
other environments provides a format to distinguish and 
contrast among them. Within an Individual environment, 
the actual values of the metrics In the set characterize a 
particular project or project subsystem. The change In 
the metric values during a project can be used to monitor 
project status and Its change over time. The characteris- 
tic set In conjunction with historical data can be used to 
forecast the outcome of the current project relative to 
past project outcome. 

The goals for this study are threefold. I.) Develop 
an approach for customizing a set of metrics to particular 
cost/quallty goals In a specific environment. II.) Apply 
the approach to calculate the characteristic set for the 
NASA/SEL environment. III.) Examine the usability of 
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the approach as a management tool for predicting out- 
come of system parts. 

3. Approach for Set Calculation 

A proposed approach for calculating a characteris- 
tic metric set consists of three steps: 1) formulate the 
goals and questions that represent cost/quallty factors In 
an environment; 2) list all metrics that capture Informa- 
tion relating to the goals and questions; and 3) condense 
metrics Into a set capturing distinct factors. This ap- 
proach satisfies the two key aspects of customizing a 
characteristic metric set to an environment: sensitivity to 
the cost/quallty goals of Importance In the environment, 
and capturing the features that give the environment Its 
Identity. 

The first step Is to generate a cost/quallty goal 
and question framework for the environment ou which to 
base the generation of all potential metrics (see Figure 1). 
After the goals and questions have been specified for an 
environment, all possible metrics are listed that represent 
relevant Information. These first two steps are an appli- 
cation of Che goal-questlon-metrlc paradigm [6, 7|. Since 
a software environment Is In some sense defined by the 
projects it develops, applying the metrics listed to those 
projects reflects an environment’s Identifying features. 
The third step Is to condense the collection of candidate 
metrics Into a characteristic set. Factor analysis may be 
applied to accomplish this step [22, 24]. This data reduc- 
tion task actually groups the metrics listed according to 
how they relate to the distinct factors In an environment. 
Appropriate metrics that relate to each of the factors can 
then be selected based on some criteria, such as ease of 
calculation or phase availability. In very heterogeneous 
environments, cluster analysis [16, 24] may first be used 
to Identify demographlcaily similar projects or subsys- 
tems, followed by factor analysis within the groups. Sec- 
tion 4.3, “NASA/SEL Set Calculation,” describes the ap- 
plication of these steps In a software production environ- 
ment. 

3.1. An Alternate Approach 

An alternate approach to determining a small set 
of characteristic metrics was examined In [35]. In this ap- 
proach, twenty candidate complexity metrics were calcu- 
lated on 585 PL/I procedures. The name of each pro- 
cedure was put Into a large “complexity pot” once for 
each time the procedure appeared In the top decile of a 
candidate complexity metric. Since there were twenty 
candidate metrics, the name of a given procedure could 
then appear up to twenty times In the pot. The pro- 
cedures Identified by a single metric were then compared 
with those In the total pot. For each appearance of a 
procedure name In the total pot, a candidate metric was 
awarded one point If that name was In the metric’s top 
decile. The candidate complexity metric that scored the 
highest would be selected for the characteristic set. All 
occurrences of procedure names were then removed from 
the pot that appeared In the top decile of the first metric 
selected. The scores for the metrics were then recalculat- 
ed based ou the remaining procedures and another metric 
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would then be selected, continuing until no procedures 
remained In the pot. 

This first approach for calculating a metric set Is 
simple and straightforward. However, there are some 
drawbacks resulting from the simplicity, Including the 
technique used to select metrics for the characteristic set 
and a fundamental assumption In the calculation. Includ- 
ing a large number of highly dependent program metrics 
In the collection examined (e.g., the software “quantity” 
group of executable statements, length, volume, vocabu- 
lary, ...) Increased disproportionately the number of ap- 
pearances of routines commonly selected by that group In 
the pot of “complex” programs. It Is therefore no 
surprise that the metric that selected the greatest percen- 
tage of the appearances In the pot Is one member of the 
;; quantity" group (length). In each of the twenty pro- 
gram metrics examined, the top decile of programs was 
chosen as the most complex according to that metric. 
This decision relied on the Implicit assumption that soft- 
ware complexity Is a monotonlcally Increasing function of 
each Of the metrics. Whlnh Js t.mnhlpsnmp 

Our paper presents an approach for calculating a 
characteristic set that advances the above approach by 1) 
selecting candidate metrics based on an environment’s 
cost/quallty goals, and 2) abstracting relationships (e.g., 
correlations) among (ln)dependent metrics Into a set of 
environmental factors. The use of values of characteristic 
metrics to Identify modules with particular attributes, 
such as those of high “complexity” as was done In [15], Is 
discussed In Section 5. 

4. Application in the NASA/SEL Environment 

This section describes the NASA/SEL environ- 
ment, the data collection, and the resulting characteristic 
metric set. 

4.1. NASA/SEL Environment 

The Software Engineering Laboratory (SEL) [2, 3, 
11, 25] Is a Joint venture between the University of Mary- 
land, NASA/Goddard Space Flight Center, and Comput- 
er Sciences Corporation. The purpose of the SEL has 
been to provide an experimental database for examining 
relationships among the factors that affect the software 
development process and the delivered product. The 
software comprising the database Is ground support soft- 
ware for satellites. The six systems analyzed In this 
study consisted of 51,000 to 112,000 lines of FORTRAN 
source code, and took between 6000 and 22,300 person- 
hours to develop over a period of 9 to 21 months. There 
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are from 2(k) to 600 modules (e.g., subroutines) In each 
system and the stair size ranges from 8 to 23 people per 
project. Including the support personnel. Anywhere from 
10 to 01 percent of the source code Is reused or modllled 
from previous projects. 

4.2. Data Collection 

The data discussed In this study are extracted 
from several sources. Among the data analyzed are the 
effort to design, code, and test the various modules of the 
systems as well as the changes and faults that occurred 
during their development. Effort data were obtained 
from a collection form that Is filled out weekly by all pro- 
grammers on the project. They report the time they 
spent on each module In the system partitioned Into the 
phases of design, code, and test, as well as any other time 
they spend on work related to the project, e.g., documen- 
tation, meetings, etc. A module Is defined as any named 
object in the system; that is, a module Is either a main 
procedure, block data, subroutine or function. The faults 
and changes are reported on another data collection form 
that Is completed by a programmer each time a change is 
made to the system. A static code analysis program 
called SAP [14] automatically computed several of the 
static metrics examined In this analysis. 

4.3. NASA/SEL Set Calculation 

In the application of the approach In the SEL en- 
vironment, there were two major reasons to use Just six 
recent projects. First, changes and Improvements In de- 
velopment technologies and personnel tend to be reflected 
In the projects developed (as they are Intended to be). 
Therefore, the consideration of projects not recently com- 
pleted would not be representative of the current environ- 
ment. Second, several development environments do not 
have a long history of data collection. Discussing an ap- 
proach that required a large project database would have 
little utility for them. 

Three goal areas were defined for the SEL environ- 
ment. The first goal area was to analyze the system de- 
velopment effort. An example question under this goal Is 
“What are the attributes of modules that result In high 
development effort?". The second goal area was to 
analyze the system modifications. An example question 
here Is “What are the attributes of modules that will be 
difficult to change?". Analyzing the system faults was 
the third goal area. An example question would be 
“What are the attributes of modules that will be Tault- 
prone?". The generated list of metrics based on these 
three goal areas appears In Table 1; a total of 40 metrics 
was examined. The metrics are grouped according to the 
general areas of slze/cornplexlty [21], effort, 
faults/changes, and software science [19]. The set- nota- 
tion In the table signifies the ratio of one metric over 
another, e.g., amount* of code effort was considered alone 
and divided by the amount of testing effort, overhead 
effort, and total effort. In addition to being examined 
alone, several effort and faults/changes metrics were di- 
vided by slze/cornplexlty metrics. 

From the six projects, this analysis focuses on 652 


Table 1. List of measures examined In the SEL 
environment. 

Slze/Onnplexlty Area 

source lines (SUC) 

executable statements (XQT) 

comments 

comments/SRC 

Cyclomatlc_complexlty 

calls 

{Cyclomatlc_complexlty} over {XQT} 

Effort Area 

total_effort 
designee ffort 
code_effort 
testlng_effort 

{deslgn_effort} over {code_effort} 

{code_effort} over 

{testlng_effort, overhead_efTort, total_effort} 
{designee ffort, code_effort, testlng_effort} over {calls} 
{designee (Tort, code_effort} over {t 7 2 } 

{total_effort} over 

{SRC, SRC-comments, XQT, calls} 

Fanlts/Changes Area 

version 

total_changes 

welghted_changes 

total_faults 

welghted_faults 

{totaM'aults. welghtcd_faults} over {SRC, XQT} 


Software Science 
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newly developed modules with complete data for the 
metrics listed In Table 1. The use of principal factor 
analysis (with orthogonal varlmax rotation) [22, 24] Iso- 
lated a set of six distinct factors, {size, modification arid 
fault correction effort density, development effort, code 
and test effort, r? 2 *, # versions}, which arc listed lri des- 
cending order of overall Importance and cumulatively ex- 
plained 79% of the variance. The t/ 2 metric Is the 
number of I/O parameters In a module. Some appropri- 
ate metrics that related well to each of the factors In the 
set were a) size - source lines, executable statements, and 
N (the total number of operators and operands): b) 
modification and fault correction effort density - fault 
correction effort / executable statement; c) development 
effort - design effort, total effort / executable statement, 
and design effort / subroutine call; d) code and test effort 
- code effort, code effort / subroutine call, and test effort 
/ subroutine call; e) r/ 2 * - Vv* and r) # vers Jons - number 
of module versions. Thus, a feasible characteristic metric 
set for the SEL environment Is {source lines, fault correc- 
tion effort pCr executable statement, design effort, code 
effort, number of I/O parameters, number of versions). 
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Table 2. Fraction of past SEL modules In the 

upper quartlle of the 

dependent variables. ! 

2a.) Module Development Effort 


Characteristic 

Quartlle of Metric M* 

Set Metric M, 

Upper 

Second 

Third 

Lower 

code effort 

.7't 

.18 

.04 

.04 

design effort 

.56 

.18 

.13 

.13 

source lines 

.51 

.20 

.14 

.09 

vi 

.48 

.24 

.17 

.11 

version 

.44 

.37 

.13 

.00 

fault correction 

.41 

.28 

.15 

.10 

effort / XQT 





2b.) Module Modification Effort 

Characteristic 

Quartlle of Metric M s 

Set Metric M s 

Upper 

Second 

Third 

Lower 

fault correction 

.05 

.18 

.08 

.09 

effort / XQT 





version 

.52 

.33 

.11 

.04 

code effort 

.50 

.27 

.17 

.00 

source lines 

.50 

.28 

.13 

.09 

vi 

.45 

.24 

.23 

.08 

design effort 

.41 

.25 

.18 

.17 

j 2c.) Module Fault Correction Effort 

Characteristic 

Quartlle of Metric M* 

Set Metric M i 

Upper 

Second 

Third 

Lower 

fault correction 

.81 

.19 

.00 

.00 

effort / XQT 





version 

.50 

.35 

.12 

.03 

code effort 

.48 

.29 

.15 

.08 

source lines 

.42 

.33 

.14 

.11 

*h' 

.42 

.28 

.19 

.11 

design effort 

.30 

.25 

.20 

.19 


5. Use as a Management Tool 

Although a characteristic set has the several uses 
outlined In Section 2, this study focuses on the use of 
metrics In the set to forecast the outcome of modules In 
projects. Several studies have pointed to the unsatisfac- 
tory use of metrics as direct predictors of software cost 
and quality [5, 20, 26). This inadequacy motivates the 
use of software metrics from a new perspective - the ex- 
amination of how well the metrics In the characteristic 
set can Identify system parts (or whole systems) resulting 
In high or low cost/quallty. System parts with Interest- 
ing cost or quality attributes Include those with high/low 
development effort, high/low modification effort, or 
high/low fault correction effort. 

An approach for using metrics to Identify system 
parts having Interesting attributes Is as follows. First, 
select some Interesting cost or quality aspect of a system 
part, such as the total development effort for a module. 
Then, choose a set of modules that would be useful to 
Identify, such as those modules that might eventually be 
in a project’s upper quartlle or total development effort. 
Next, from past projects determine how often metric 


value ranges (c.g., quartlles) contained modules that end- 
ed up In the upper quartlle of development effort. Final- 
ly, characterize and Identify modules In a current project 
that are likely, based on past metric data, to end up In 
the upper quartlle of total development effort. The calcu- 
lation of a characteristic metric set and the use of 
corresponding metric data from past projects Is Intended 
to help Identify Interesting modules In a system. 

5.1. Metric Data from Past Projects 

The data displayed In Table 2 were calculated 
from six SEL projects, and are Interpreted as follows. 
The table Is divided Into three sections, corresponding to 
the three SEL goal areas discussed above. There Is a 
table section for each dependent variable: total module 
development effort, total effort for module modification, 
and total effort for fault correction In a module. The 
characteristic set of six metrics that represent the 
different environmental factors Is listed In each section of 
the table. Consider the section on total module develop- 
ment effort. The table displays the fraction of modules 
contained In the upper quartlle of total development 
effort, based on their final quartlle rankings for the 


Table 3. Fraction of past SEL modules In the , 

I lower quartlle of the 

dependent variables. 1 

3a.) Module Develop ment 

rr>pr~ 

UiiUi 


Characteristic 

Quartlle of Metric M, 

Set Metric M f 

Upper 

Second 

Third 

Lower 

code effort 

.00 

.00 

.23 

.77 

source lines 

.10 

.12 

.24 

.54 

version 

.06 

.14 

.30 

.50 

vi 

.09 

.21 

.25 

.45 

design effort 

.02 

.23 

.37 

.38 

fault correction 

.12 

.25 

.32 

.31 

effort / XQT 





3b.) Module Modification Effort 

Characteristic 

Quartlle of Metric M, 

Set Metric Mj 

Upper 

Second 

Third 

Lower 

version 

.09 

.15 

.28 

.48 

fault correction 

.01 

.13 

.43 

.43 

effort / XQT 





vi 

.14 

.19 

.25 

.42 

source lines 

.11 

.18 

.30 

.41 

code effort 

.11 

.18 

.34 

.37 

design effort 

.18 

.28 

.27 

.27 

3c.) Module Fault Correction Effort 

Characteristic 

Quartlle of Metric M, 

Set Metric 

Upper 

Second 

Third 

Lower 

fault correction 

.00 

.oo 

.50 

.50 

effort / XQT 





version 

.18 

.18 

.27 

.37 

source lines 

.21 

.19 

.29 

.31 

code effort 

.18 

.24 

.27 

.31 

Vi 

.20 

.24 

.25 

.31 

design effort 

.18 

.25 

.29 

.28 
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characteristic metrics. For example, 71% of the modules 
In the upper quartlle of code effort were also In the upper 
quartlle of total module development clTort. Only 9% of 
the modules In the lower quartlle of source lines were In 
the upper quartlle of total development effort. The In- 
terpretation Is the same for the other dependent variables 
of module modification effort and module fault correction 
effort. Table 3 Is analogous to Table 2, except It displays 
the fraction of modules contained In the lower (Instead of 
the upper) quartlle or the respective dependent variable. 
For example, 50% of the modules In the lower quartlle of 
number of versions were also In the lower quartlle of total 
module development effort. 

5.2. Data Interpretation 

The Information In these tables could be used to 
forecast the outcome of modules In a system. At the end 
of the design phase, the r/ 2 metric and the amount of 
effort spent In design are known. The modules In the 
upper quartlle of design effort should be Identified by a 
project manager because 56% of these modules ended up 
in the upper quartlle of total development effort. That Is, 
In this environment the modules In the upper quartlle of 
design effort were more than twice (=.56/.25) as likely 
than by chance to be the most expensive to develop 
overall; these modules were approximately 28 (—.50/.02) 
times more likely to be In the upper quartlle of total de- 
velopment effort than to be In the lower quartlle of total 
development effort. Modules In the upper quartlle of the 
r/ 2 * metric were almost twice as likely than by chance to 
require the most effort to develop, modify, and correct. 
Other observations Include 1) It Is easiest to Identify 
those modules that will have high development effort; 2) 
It Is most difficult to Identify those modules that will re- 
quire little fault correction effort; and 3) the metrics of 
design effort and 77 2 are reasonably similar In forecasting 
ability, except that ? 7 2 seems superior In Identifying 
modules that will require little modification effort. 

The two tables help characterize the SEL develop- 
ment environment. The total development effort for a 
module tends to be Indicated by the module’s coding 
effort - modules In the extreme quartlles of coding effort 
are three times more likely than by chance to be In the 
corresponding extreme quartlles of total development 
effort. Since the programmers In the SEL are quite ex- 
perienced In the application area and with appropriate 
design approaches, the dominance of coding effort seems 
reasonable. In other environments, the amount of design 
effort might better Indicate the total development effort 
required. Other observations Include 1) high density of 
fault correction effort (fault correction effort per execut- 
able statement) Indicates high total modification- effort 
and high total fault correction effort; and 2) an extreme 
(high or low) number of program versions reflects a 
corresponding amount of modification effort and correc- 
tion effort. 

Ideally, the metrics In the characteristic set would 
all be available early In development and have strong re- 
lationships with the dependent variables of Interest. 
Some metrics, such as fault correction .effort per execut- 


able statement, have limited usefulness ;is a predictor be- 
cause of not being available until late In project develop- 
ment. An assumption Is needed In order to use metric 
data from past projects to forecast the outcome of 
modules from a current project. The assumption Is that 
the relationship between a module’s current metric quar- 
tlle and Its eventual outcome (l.e., development, 
modification, and correction effort) Is the same as the re- 
lationship between the Anal metric quartlles of past pro- 
jects* modules and their outcome.' This assumption Is 
reasonable when using data from recent projects that are 
similar to the current project, and when predicting from 
metrics whose Anal quartlles are reasonably certain early 
In development (e.g„ the number of I/O parameters In a 
module tends to remain relatively constant once specified 
In the design phase, and therefore, the metric’s value does 
not tend to change quartlles). Note that the examples 
and metric data presented are from a particular environ- 
ment, project data from other environments may differ. 

Using a characteristic metric set with correspond- 
ing data from past projects enables the monitoring of a 
small set of customized metrics to forecast current project 
outcome. A characteristic set Is usable as a management 
tool as soon as the metrics In the set are available. 

6. Conclusions 

A characteristic software metric set Is Intended to 
help support the effective management of software devel- 
opment and maintenance. The approach examined for 
building a characteristic metric set Is adaptable to 
different cost/quality goats and to different environments. 
The calculation and !i«5A of t.hA t. non Id Ua Ar».ir>lAr1 try vm 
automated project monitor and database. The major 
results of this study are l) an approach has been 
described for customizing a characteristic software metric 
set to an environment: 2) the application of the approach 
to the SEL production environment yielded the charac- 
teristic software metric set {source lines, fault correction 
effort per executable statement, design effort, code effort, 
number of I/O parameters, number of versions}; and 3) 
the use of a characteristic metric set with corresponding 
historical data can assist In project management by fore- 
casting the outcome of system parts. 

Further Investigation In this area Includes Incor- 
porating the characteristic metric set data (from Section 
5) Into a knowledge-based system. A statistical pattern 
classification scheme [23| is under consideration, although 
such an approach applies Bayes* Theorem and would as- 
sume Independence among the metrics In the characteris- 
tic set. In this environment Independence between, for 
example, design effort and number of I/O parameters Is 
reasonable, while Independence between source lines and 
code effort Is questionable. A knowledge-based system 
that could use Information from several metrics simul- 
taneously would characterize system parts more 
effectively and forecast their * outcomes more precisely. 
This work Is Intended to advance the understanding of 
the use of various metrics to characterize and predict as- 
pects of software cost and quality. 
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